Sphere-covering, measure concentration, and source coding

Suppose A is a finite set, let P be a discrete distribution on A, and let M be an arbitrary “mass” function on A. We give a precise characterization of the most efficient way in which An can be almost-covered using spheres of a fixed radius. An almost-covering is a subset Cn of An, such that the union of the spheres centered at the points of Cn has probability close to one with respect to the product distribution Pn. Spheres are defined in terms of a singleletter distortion measure on An, and an efficient covering is one with small mass M(Cn). In information-theoretic terms the sets Cn are rate-distortion codebooks, but instead of minimizing their size we seek to minimize their mass. With different choices for M and the distortion measure on A our results give various corollaries as special cases, including Shannon’s classical rate-distortion theorem, a version of Stein’s lemma (in hypothesis testing), and a new converse to some measure-concentration inequalities on discrete spaces. Under mild conditions, we generalize our results to abstract spaces and non-product measures. Keywords— Sphere covering, measure-concentration, data compression, large deviations.